E cient Locally Weighted Polynomial Regression Predictions

Locally weighted polynomial regression LWPR is a popular instance based al gorithm for learning continuous non linear mappings For more than two or three in puts and for more than a few thousand dat apoints the computational expense of pre dictions is daunting We discuss drawbacks with previous approaches to dealing with this problem and present a new algorithm based on a multiresolution search of a quickly constructible augmented kd tree Without needing to rebuild the tree we can make fast predictions with arbitrary local weight ing functions arbitrary kernel widths and arbitrary queries The paper begins with a new faster algorithm for exact LWPR predictions Next we introduce an approx imation that achieves up to a two orders of magnitude speedup with negligible accu racy losses Increasing a certain approxi mation parameter achieves greater speedups still but with a correspondingly larger accu racy degradation This is nevertheless useful during operations such as the early stages of model selection and locating optima of a t ted surface We also show how the approx imations can permit real time query speci c optimization of the kernel width We con clude with a brief discussion of potential ex tensions for tractable instance based learning on datasets that are too large to t in a com puter s main memory Locally Weighted Polynomial Regression Locally weighted polynomial regression LWPR is a form of instance based a k a memory based al gorithm for learning continuous non linear mappings from real valued input vectors to real valued output vectors It is particularly appropriate for learning com plex highly non linear functions of up to about in puts from noisy data Popularized in the statistics literature in the past decades Cleveland and Delvin Grosse Atkeson et al a it is en joying increasing use in applications such as learn ing robot dynamics Moore Schaal and Atke son and learning process models Both classi cal and Bayesian linear regression analysis tools can be extended to work in the locally weighted frame work Hastie and Tibshirani providing con dence intervals on predictions on gradient estimates and on noise estimates all important when a learned mapping is to be used by a controller Atkeson et al b Schneider Let us review LWPR We begin with linear regression on one input and one output Global linear regression left of Figure nds the line that minimizes the sum squared residuals If this is represented as